D7.4 Third evaluation report. Evaluation of PANACEA v3 and produced resources

D7.4 reports on the evaluation of the different components integrated in the PANACEA third cycle of development as well as the final validation of the platform itself. All validation and evaluation experiments follow the evaluation criteria already described in D7.1. The main goal of WP7 tasks was to test the (technical) functionalities and capabilities of the middleware that allows the integration of the various resource-creation components into an interoperable distributed environment (WP3) and to evaluate the quality of the components developed in WP5 and WP6. The content of this deliverable is thus complementary to D8.2 and D8.3 that tackle advantages and usability in industrial scenarios. It has to be noted that the PANACEA third cycle of development addressed many components that are still under research. The main goal for this evaluation cycle thus is to assess the methods experimented with and their potentials for becoming actual production tools to be exploited outside research labs. For most of the technologies, an attempt was made to re-interpret standard evaluation measures, usually in terms of accuracy, precision and recall, as measures related to a reduction of costs (time and human resources) in the current practices based on the manual production of resources. In order to do so, the different tools had to be tuned and adapted to maximize precision and for some tools the possibility to offer confidence measures that could allow a separation of the resources that still needed manual revision has been attempted. Furthermore, the extension to other languages in addition to English, also a PANACEA objective, has been evaluated. The main facts about the evaluation results are now summarized

An automata based approach to biomedical named entity recognition

ing an automata learning algorithm: Causal-State Splitting Reconstruction [1]. This algorithm has previously been applied to Named Entity Recognition [2] obtaining good results given the simplicity of the approach. The same approach has been applied to Biomedical NE identification, using GENIA corpus 3.0, with 10-fold cross-validation. Our system attained F1 = 73.14%. These results can be compared directly to [3] and [4], which used the same data. First system obtains F1 = 57.4% using ME Models, and the second one reports F1 = 79.2% using SVMs. Both improve their results using post-processing techniques, reaching F1 = 76.9% and F1 = 79.9% respectively. Our system does not use any post-processing techniques, and takes into acount few features, so the results are considered very promising. In future work some post-processing will be developed to improve the results